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GOVERNMENT  RIGHTS  IN  DATA  STATEMENT 


Reproduction  of  this  publication  in  whole 
or  part  is  permitted  for  any  purpose 
of  the  United  States  Government. 


EXECUTIVE  SUMMARY 


This  document  defines  the  Verification,  Validation,  and  Accreditation  (VV&A) 
process  for  Models  and  Simulations  (M&S)1  used  by  the  Single  Integrated  Air  Picture 
(SIAP)  System  Engineering  Task  Force  (SE  TF).  It  specifies  the  participants,  products, 
and  process  necessary  to  achieve  accreditation. 

The  SIAP  SE  TF  employs  an  M&S  accreditation  process  that  complies  with 
Department  of  Defense  (DoD)  guidance  for  VV&A.  The  process  is  designed  to 
accurately  determine  the  ability  of  each  designated  M&S  to  fulfill  its  specific  intended 
use  through  a  defined  series  of  plans  and  reports. 

The  VV&A  process  has  4  goals: 

1 .  Documentation  of  Capabilities,  Limitations  and  Assumptions 

2.  Integration  with  test  planning 

3.  Accreditation  before  test 

4.  Objective  assessments 

This  document  also  provides  guidance  on  the  VV&A  for  federations  of  M&S. 


1  The  acronym  “M&S”  includes  modeling  and  simulation,  model  and  simulation,  and  models  and  simulations. 
Within  this  instruction,  “M&S”  will  be  used  interchangeably  with  these  various  meanings  as  applicable. 
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1.  INTRODUCTION 


1.1  PURPOSE 

This  document  defines  the  Verification,  Validation,  and  Accreditation 
(VV&A)  process  for  Models  and  Simulations  (M&S)  used  by  the  Single  Integrated 
Air  Picture  (SIAP)  System  Engineering  Task  Force  (SE  TF).  It  describes  the 
rationale  and  objectives  for  a  VV&A  process,  as  well  as  specifies  the  participants, 
products,  and  process  necessary  to  achieve  M&S  accreditation. 

Accreditation  establishes  the  M&S  as  acceptable  for  specific  uses  in  the 
design,  development,  test,  and  evaluation  of  the  SIAP.  M&S  accreditation  is 
accomplished  in  response  to  specific  user  requirements,  and  via  a  documented 
combination  of  Verification  and  Validation  (V&V)  techniques. 

This  M&S  accreditation  process  is  tailored  to  the  needs  of  the  SIAP  SE 
TF,  but  complies  with  Department  of  Defense  (DoD)  instructions  regarding  VV&A 
(References  1  through  4). 

1.2  PROGRAM  DESCRIPTION 

The  DoD  has  substantial  evidence  that  significant  warfighting  capability 
shortfalls  exist  today  in  the  Joint  counter-air  mission  area.  Because  of  the 
prevalence  of  these  shortfalls,  the  Joint  Requirements  Oversight  Council  (JROC) 
recommended  designation  of  a  lead  system  engineering  organization  to  facilitate 
the  transition  of  the  SIAP  requirement  from  concept  to  a  fielded  joint  capability. 
The  SIAP  SE  TF  is  satisfying  this  mission  by  implementing  a  disciplined  system 
engineering  process. 

The  SIAP  SE  TF  performs  systems  engineering  and  analysis  of  current, 
programmed,  and  proposed  theater  air  and  missile  defense  systems,  networks, 
interfaces,  and  Tactics,  Techniques,  and  Procedures  (TTP),  as  well  as  external 
supporting  systems  (e.g.,  global  navigation  and  time  distribution  systems).  The 
purpose  is  to  identify  and  characterize  SIAP  issues  and  shortfalls,  warranting 
correction,  and  to  assess  the  expected  operational  benefits  and  military  utility  of 
correcting  such  shortfalls.  The  SIAP  system  engineering  is  conducted  in  close 
collaboration  with  representatives  from  the  Joint  Staff,  the  Military  Services,  and 
other  DoD  Agencies. 

1.3  IMPORTANCE  OF  VV&A 

The  SIAP  SE  TF  ensures  that  all  M&S-based  analysis  tools  used  in  block 
assessments  are  subjected  to  a  VV&A  process,  which  addresses  the  suitability  of 
the  M&S  for  the  specific  analysis.  Since  much  of  the  SIAP  SE  TF’s  analysis  is 
based  on  M&S,  the  results  of  the  VV&A  process  are  a  critical  part  of  the  SIAP  SE 
TF’s  system  engineering. 


1 

April  2003 


DoD  emphasis  on  VV&A  began  in  1995  when  the  Defense  Modeling  and 
Simulation  Office  (DMSO)  initiated  a  VV&A  Program  in  response  to  a  tasking  in 
the  DoD  M&S  Master  Plan  (Reference  1).  VV&A  policy  was  established  in  the 
form  of  DoDI  5000.61,  DoD  M&S  VV&A  (Reference  2),  and  core  guidance  in  the 
form  of  the  DMSO  VV&A  Recommended  Practices  Guide  (RPG)  (Reference  3). 
DoDI  5000.61  implements  policy,  assigns  responsibilities,  and  prescribes 
procedures  for  the  VV&A  of  DoD  M&S.  As  part  of  the  DoD  policy,  M&S  used  to 
support  the  major  DoD  decision-making  organizations  and  processes  shall  be 
accredited  for  that  use  by  the  DoD  organization  sponsoring  the  application. 

VV&A  guidance  provided  in  the  Interim  Defense  Acquisition  Guidebook 
(Reference  4)  states:  “The  Program  Manager  (PM)  shall  use  verified,  validated, 
and  accredited  models  and  simulations,  and  ensure  credible  applicability  for  each 
proposed  use.” 

DoDI  5000.61  provides  the  following  definitions: 

Verification:  the  process  of  determining  that  a  model  implementation  and 
its  associated  data  accurately  represent  the  developer’s  conceptual 
description  and  specifications.  Verification  answers  the  question,  “Did  I 
build  the  thing  right?” 

Validation:  the  process  of  determining  the  degree  to  which  a  model  and  its 
associated  data  are  an  accurate  representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the  model.  Validation  answers  the 
question,  “Did  I  build  the  right  thing?” 

Accreditation:  the  official  certification  (by  the  user)  that  a  model, 
simulation,  or  federation  of  models  and  simulations  and  its  associated 
data  are  acceptable  for  use  for  a  specific  purpose.  Accreditation  answers 
the  question,  “Does  it  suit  my  needs?” 
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2.  ACCREDITATION  APPROACH 

The  SIAP  SE  TF  VV&A  process  has  the  following  goals: 

1 .  Documentation  of  Capabilities,  Limitations  and  Assumptions  (C/L/A) 

It  is  essential  to  understand  the  C/L/A  that  are  inherent  in  the  M&S  as 
they  pertain  to  the  specific  issues  in  the  analysis.  For  example,  by 
knowing  what  aspects  of  a  real-world  system  have  been  approximated  or 
are  not  addressed,  the  TF  can  assess  how  the  M&S  can  be  used  in  the 
analysis,  as  well  as  how  the  M&S  can  be  improved. 

2.  Integration  with  test  planning 

The  VV&A  process  is  integrated  into  the  test  planning  and  execution 
that  occurs  for  each  SIAP  Block  issue,  since  assessment  of  an  M&S  tool 
is  done  with  specific  reference  to  that  issue.  This  test  process  is 
described  in  the  SE  TF  Integrated  Assessment  Plan  (IAP),  Standard  Data 
Management  and  Analysis  Plan  (DMAP)  and  Standard  Test  Plan 
(References  5  through  7).  This  approach  recognizes  there  are  resource 
(time,  funding,  personnel)  limitations  that  affect  the  VV&A  process  and 
gives  V&V  issues  greater  visibility  during  test  planning,  execution  and 
reporting. 

3.  Accreditation  before  test 

The  SIAP  SE  should  accredit  the  M&S  environment  as  a  part  of  the 
Test  Plan  approval  prior  to  executing  the  test.  The  accreditation  should 
reflect  the  Task  Force’s  confidence  that  M&S  C/L/A  will  not  overshadow 
the  evaluation  results. 

4.  Objective  assessments 

Validation  by  an  independent  organization  (unrelated  to  the  M&S 
Developer  (MSD))  provides  an  objective  assessment  of  whether  the  M&S 
has  met  its  requirements.  Such  an  organization  may  be  available  through 
the  MSD’s  parent  Service/Agency,  a  Government  organization,  a 
contractor/private  organization  or  by  assembling  a  team  of  personnel  from 
within  the  SIAP  community,  Throughout  the  VV&A  process  there  are  also 
reviews  that  ensure  the  V&V  activities  have  input  from  all  interested 
Services  and  Agencies. 
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2.1  PARTICIPANTS 


An  overview  of  the  roles  and  responsibilities  of  each  of  the  SIAP  VV&A 
participants  is  provided  in  Table  1. 


Participant’s  Role 


Issue 

Lead/Team 

AA 

SAT 

ESG 

VV&A 

Action 

Team 

MSD 

Services 

Define  Problem 

Lead 

X 

X 

Define  M&S  Requirements 

Lead/Approve 

X 

X 

X 

X 

Define  Acceptance  Criteria 

Approve 

X 

X 

X 

X 

Designate  M&S 

Lead 

Approve 

Review 

X 

X 

X 

Provide  V&V  Guidance 

Lead 

X 

Develop  V&V  Plan 

X 

Approve 

Review 

Lead 

X 

Execute  V&V  Plan 

X 

X 

Lead 

X 

Develop  V&V  Report 

X 

X 

Lead 

X 

Develop  V&V  Assess  Report 

Review 

Lead 

X 

Make  Accreditation  Decision 

X 

Approve 

X 

X 

X 

X 

Compile  Accreditation  Package 

Lead 

Table  1.  VV&A  Roles  and  Responsibilities 

Table  1  roles  explanations  (from  DMSO  RPG): 

a.  Approve:  “Determines  when  an  activity  is  satisfactorily  completed  and 
another  can  begin.  Determines  what  activity  should  be  pursued  next  (e.g., 
whether  to  continue  on  to  the  next  scheduled  activity  or  to  return  to  a 
previous  activity)” 

b.  Lead:  “Leads  the  task.  Normally  involves  active  participation  from  others.” 

c.  Review:  “Participation  normally  limited  to  reviewing  results  of  task  and 
providing  recommendations.” 

d.  X  -  For  purposes  of  Table  1  within  this  document  only,  X  is  used  to  denote 
a  supporting,  assisting,  or  other  participating  role. 

2.1.1  Issue  Lead/Team 

The  Issue  Lead  with  supporting  Subject  Matter  Experts  (SMEs)  and 
Service  Representatives  will  employ  the  products  or  services  from  models, 
simulations,  or  simulation  federations  to  achieve  its  purposes.  The  Issue  Leads 
shall: 

a.  Define  the  problem  to  be  addressed  by  M&S. 

b.  Define  M&S  objectives/requirements. 

c.  Define  acceptance  criteria. 

d.  Designate  the  M&S  they  will  use  to  analyze  their  issue  in  the  IAP. 
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2.1.2  Accreditation  Authority 

The  AA  is  the  SIAP  SE,  unless  delegated  to  the  Deputy  SE  or  Technical 
Director.  The  AA  shall: 

a.  Approve  the  M&S  designation  (through  the  IAP  or  a  Designation  Letter). 

b.  Accredit  the  M&S. 

2.1.3  SIAP  Analysis  Team  Executive  Steering  Group  (SAT  ESG) 

The  SAT  ESG,  per  the  SAT  Charter  (Reference  8),  is  a  permanent  body 
that  provides  overarching  support  for  planning,  execution,  analysis,  and  reporting 
of  SIAP  analysis  activities.  The  membership  includes  the  SIAP  SE  TF  M&S 
Chief,  Service  representatives,  Issue  Leads,  and  SMEs.  The  SAT  ESG  shall: 

a.  Review  the  lAP/Letter  of  Designation. 

b.  Approve  the  V&V  Plan. 

c.  Review  the  V&V  Assessment  Report  and  provide  an  accreditation 
recommendation  to  the  AA. 

2.1.4  VV&A  Action  Team 

A  VV&A  Action  Team,  per  the  SAT  Charter,  is  an  ad  hoc  team  of  SMEs, 
Model/Tool  experts,  Service  representatives  and  other  specialists.  Its 
membership  varies  based  on  the  specific  objectives  and  M&S  required,  and  it  will 
normally  be  established  as  part  of  the  Test  Plan  Working  Group  activities.  The 
Action  Team  is  organized  and  led  by  the  VV&A  Activity  Manager,  who  is  the 
SIAP  SE  TF  M&S  Chief.  The  M&S  Chief  works  with  Block  Issue  Leads,  SMEs, 
MSDs,  the  SAT  ESG,  and  the  AA  to  ensure  the  VV&A  process  meets  its 
objectives.  The  Action  Team  shall: 

a.  Provide  guidance  and  resolve  issues  with  the  MSD  throughout  the  VV&A 
Process. 

b.  Designate  M&S  with  a  Designation  Letter  (if  not  done  in  the  IAP). 

c.  Review  the  V&V  Plan  and  provide  an  approval  recommendation  to  the 
SAT  ESG. 

d.  Assess  the  V&V  results  and  make  an  accreditation  recommendation  to  the 
SAT  ESG. 

e.  Compile  the  Accreditation  Package  (M&S  Chief  only). 

2.1.5  M&S  Developer  (MSD) 

The  MSD  is  the  organization  that  is  responsible  for  developing  and 
directing  configuration  control  of  a  model  or  simulation.  The  MSD  develops  and 
executes  a  strategy  for  developing  and  maintaining  the  M&S  throughout  its  life 
cycle. 
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The  MSD  shall: 


a.  Write  the  V&V  Plan  (as  an  Appendix  to  the  Test  Plan). 

b.  Execute  the  V&V  Plan. 

c.  Write  the  V&V  Report  (as  an  Appendix  to  the  Test  Plan). 

2.1.6  Services 

The  Services  provide  representatives  and/or  SMEs  who  participate  in  the 
VV&A  process.  These  Service  personnel  shall: 

a.  Assist  the  Issue  Leads  in  defining  objectives/requirements,  acceptance 
criteria  and  designating  the  M&S. 

b.  Assist  as  members  of  the  VV&A  Action  Team. 

2.2  PRODUCTS 

The  M&S  VV&A  process  is  marked  by  documentation  that  records  all 
accreditation  phases.  The  specific  documents,  or  products,  associated  with  the 
accreditation  process  are  discussed  in  the  following  paragraphs. 

2.2.1  M&S  VV&A  Designation  Document 

The  intent  to  use  a  particular  M&S  for  a  specific  analysis  is  determined 
through  a  deliberate  process,  documented,  and  then  approved  by  the  AA.  In  the 
VV&A  process,  there  are  two  types  of  Designation  Documents.  The  first  is  the 
IAP  (Reference  5).  The  Block  Issue  Leads  normally  create  or  update  their 
issue’s  section  in  the  IAP  (Appendix  C  for  FY02-03  efforts)  prior  to  test  execution, 
with  the  IAP  being  approved  by  the  SIAP  SE  or  their  designee.  Since  the 
information  in  the  IAP  covers  the  use  of  a  particular  M&S  for  each  issue, 
designation  of  the  M&S  occurs  with  approval  of  the  IAP  by  the  SIAP  SE/AA. 

The  second  type  of  Designation  Document  is  the  Designation  Letter  (see 
Appendix  A-2  for  a  sample  format).  The  VV&A  Action  Team  writes  the 
Designation  Letter  when  it  is  necessary  to  separate  the  designation  decision  from 
IAP  updates  to  meet  schedule  demands  or  other  needs.  It  notifies  the  MSD  of 
the  AA’s  need  to  conduct  VV&A  on  the  uniquely  identified  M&S  and  directs  the 
MSD’s  support  of  this  effort.  The  AA  or  their  designee  signs  the  letter. 

2.2.2  V&V  Plan 

The  V&V  Plan,  developed  by  the  MSD,  details  the  V&V  methodologies 
that  shall  be  used  by  the  MSD  to  establish  the  suitability  of  the  M&S  for 
accreditation.  For  each  M&S  requirement,  the  Plan  defines  the  set  of  procedures 
intended  to  generate  the  data  necessary  for  an  assessment  of  M&S  capabilities 
and  limitations.  The  V&V  Plan  is  an  Appendix  of  the  Test  Plan,  and  it  should 
address  the  following  areas: 
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2.2.2. 1  M&S  Requirements/Acceptance  Criteria 

What  are  the  requirements  guiding  the  analysis  and  the  specific  use  of  the 
M&S,  as  defined  during  the  Designation  Process?  How  will  V&V  assess  the 
capability  of  the  M&S  to  address  these  requirements,  i.e. ,  what  are  the 
acceptance  criteria  for  the  M&S?  What  capability  must  be  demonstrated  for  the 
M&S  to  be  viewed  as  sufficiently  credible  for  the  AA  to  accredit? 

2. 2. 2. 2  Capabilities/Limitations/Assumptions  (C/L/A) 

What  aspects  of  the  problem  or  system  can  and  cannot  be  modeled  that 
affects  the  use  of  the  M&S  in  this  specific  analysis?  What  limits  are  there  on 
what  data  can  be  used?  What  simplifying  assumptions  were  made  regarding 
operational  systems  and  scenarios  (i.e.,  perfect  connectivity  or  sensor 
capabilities),  or  the  method  of  implementing  applicable  MIL-STDs? 

2. 2. 2. 3  V&V  Methods 

Describe  the  specific  techniques  that  will  be  used.  The  V&V  execution 
should  consider  use  of  the  V&V  methods  defined  in  the  DMSO  RPG,  which 
provides  information  on  over  75  V&V  techniques. 

2. 2.2. 4  Data  Certification 

Document  who  provides  the  data,  scenario  or  other  input  to  the  M&S,  and 
summarize  how  they  certify  it  has  been  correctly  recorded  and/or  is  operationally 
valid.  This  is  especially  important  for  data  from  “live”  events  and  Common 
Reference  Scenarios. 

2. 2. 2. 5  M&S  Development  Methodology 

Is  this  a  legacy  model  with  no  changes  and  a  well-documented  history? 

Or  a  legacy  model  that  will  be  changed?  Or  is  it  completely  new?  How  are 
changes/new  development  achieved — using  spirals  or  single  step  capability? 

2. 2. 2. 6  Configuration  Management  Plan 

An  important  factor  for  the  accreditation  of  M&S  is  sound  Configuration 
Management  (CM)  practices  in  the  development  of  the  M&S.  The  CM  Plan, 
developed  and  executed  by  the  MSD,  documents  the  configuration  control 
process  for  the  designated  M&S.  The  CM  Plan  is  normally  a  stand-alone 
document,  but  is  referenced  in  the  V&V  Plan.  This  is  especially  important  if  the 
M&S  will  be  changed/upgraded  during  the  course  of  the  test.  Section  1 .6.3  of  the 
DMSO  RPG  provides  additional  information  on  CM. 
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2.2.3  V&V  Report 


The  V&V  Report,  generated  by  the  MSD,  describes  the  results  of  the  V&V 
execution  and  explains  test  anomalies  or  the  inability  of  the  M&S  to  meet  the 
acceptance  criteria.  It  is  an  Appendix  to  the  Test  Plan. 

2.2.4  V&V  Assessment  Report 

The  V&V  Assessment  Report  is  a  memorandum  that  summarizes  the 
overall  V&V  execution,  provides  an  official  assessment  of  the  extent  that  the 
collected  data  supports  the  V&V  requirements,  and  makes  a  recommendation  on 
accrediting  the  M&S  for  the  specific  analysis.  It  is  generated  by  the  VV&A  Action 
Team  and  reviewed  by  the  SAT  ESG. 

2.2.5  Accreditation  Package 

The  Accreditation  Package  is  the  complete  set  of  unclassified  VV&A 
documents  for  a  specific  analysis,  and  it  is  posted  on  the  SIAP  SE  TF  Worksite 
for  reference  on  future  Block  analyses  and  access  by  Issue  Leads. 
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2.3  PROCESS 


The  SIAP  M&S  VV&A  Process  is  divided  into  four  phases:  designation, 
planning,  V&V  execution,  and  accreditation  (see  Figure  1). 


Figure  1.  SIAP  M&S  VV&A  Process 


2.3.1  Phase  1:  Designation 

Designation  occurs  when  the  intent  to  use  a  particular  M&S  for  a  specific 
analysis  is  documented  in  writing  and  approved  by  the  AA  or  their  designee. 
This  will  be  done  in  either  the  IAP  or  a  Designation  Letter  (see  Section  2.2.1 ). 


2. 3. 1.1  Designation  Process 

The  designation  process  establishes  the  direction  and  pace  of  the 
resulting  VV&A  effort.  It  is  a  critical  factor  in  the  success  of  that  effort.  When 
possible,  the  SIAP  SE  TF  follows  a  five-step  designation  process  that  allows  the 
SIAP  SE  TF  and  the  MSD  to  agree  that  the  M&S  selected  is  capable  of  satisfying 
the  specified  need,  and  there  are  sufficient  resources  to  complete  accreditation. 
The  designation  process  will  minimize  any  surprises  with  the  M&S  at  the  end  of 
the  accreditation  process,  and  provides  a  justification  as  to  why  one  M&S  may  be 
selected  over  another.  The  Issue  Lead(s)  with  the  supporting  SMEs  and  Service 
Representatives  on  their  team  normally  execute  this  process,  with  support  from 
the  VV&A  Action  Team,  MSD,  and  SAT  ESG. 

1 .  Identify  need 

Define  the  specific  intended  use  for  the  M&S  and  establish  the 
necessity  of  accreditation  based  on  what  kind  of  data  is  needed  to  conduct 
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analysis  of  the  SIAP  issue. 


2.  Establish  objectives/requirements 

Document  functionality  requirements  to  support  the  specific  intended 
use.  Consider  what  level  of  fidelity  must  be  demonstrated  to  provide  data 
of  sufficient  accuracy  for  the  analysis. 

3.  Understand  M&S  functionality 

Document  the  current  M&S  capabilities  that  appear  to  be  able  to  meet 
the  requirements  This  will  require  discussions  with  MSDs  to  determine  if 
there  are  several  M&S  that  can  do  the  job,  or  none.  It  may  be  important  to 
understand  the  software  development  capabilities  of  the  MSD,  to  include 
any  maturity  levels  achieved  (such  as  from  the  Software  Engineering 
Institute  Capability  Maturity  Model)  as  well  as  previous  V&V  efforts  and 
M&S  experience.  It  may  also  be  necessary  to  improve  the  capabilities  of 
M&S  tools  in  order  for  it  to  be  useful  in  the  specific  intended  use. 

4.  Conduct  a  Front-end  Assessment  of  Alternatives 

Identify  the  programmatic  issues  (schedule,  cost,  risk)  for  each  M&S 
alternative  being  considered. 

5.  Make  Designation  Decision 

Use  technical  and  programmatic  assessments  to  designate  the  “best 
choice”  M&S  for  accreditation. 

2.3.2  Phase  2:  Planning 

The  planning  phase  of  the  VV&A  process  begins  with  the  development  of 
the  V&V  Plan  by  the  MSD.  The  plan  contains  the  specific  qualitative  and/or 
quantitative  testing  requirements  to  satisfy  the  acceptance  criteria.  V&V  Plans 
may  vary  greatly  based  upon  previous  V&V  efforts,  complexity  of  simulation 
functionality,  length  of  use,  scope  of  intended  use,  and  clarity  of  application  area 
requirements.  The  MSD  should  consider  use  of  the  various  V&V  techniques 
defined  in  the  DMSO  RPG,  and  work  with  the  VV&A  Action  Team  and  Block 
Issue  Lead(s)  to  provide  a  Plan  that  balances  analysis  objectives  with  resources 
(time,  funding,  personnel),  risk,  Service/Agency  M&S  directives  and  their  own 
organizational  culture/experience. 

The  VV&A  Action  Team  assesses  the  V&V  report  and  makes  an  approval 
recommendation  to  the  ESG.  At  this  point  in  the  planning,  it  is  expected  that  the 
Test  Plan  will  still  be  a  “draft”  document,  but  the  V&V  Plan  should  be  submitted 
as  a  “final”  version.  This  requires  V&V  activities  be  addressed  early  in  the  test 
planning.  The  draft  Test  Plan  should  be  available  to  ensure  the  V&V  Plan  is 
reviewed  in  context.  The  ESG  then  reviews  the  V&V  Plan  and  decides  if  it  is 
acceptable  (approves)  or  recommends  changes  (disapproves). 
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Reviewing  the  V&V  plan  early  in  the  test  planning  process  results  in 
significant  risk  reduction  for  the  test  readiness  and  accreditation  decision  by 
providing  the  SIAP  SE  TF  and  Services/Agencies  insight  on  the  test  plans.  ESG 
approval  of  the  V&V  plan  should  not  be  construed  as  “guaranteeing” 
accreditation — it  is  intended  to  be  part  of  a  collaborative  effort  between  the  SIAP 
SE  TF,  the  MSD  and  Service  SMEs  during  every  step  of  test  planning  and 
execution. 

2.3.3  Phase  3:  V&V  Execution 

The  VV&A  Action  Team  monitors  V&V  execution  progress  and  addresses 
issues  or  concerns.  When  all  required  V&V  efforts  and  documentation  is 
complete,  the  MSD  provides  the  V&V  Report  to  the  VV&A  Action  Team  for 
evaluation.  The  latest  version  of  the  Test  Plan  should  be  available  to  ensure  the 
V&V  Report  is  reviewed  in  context. 

The  V&V  Report  summarizes  all  V&V  efforts  in  accordance  with  the 
requirements  set  forth  in  the  V&V  Plan.  The  VV&A  Action  Team  evaluates  the 
report  for  consistency,  correctness,  and  completeness  and  makes  a 
recommendation  to  the  ESG  via  the  VV&A  Assessment  Report.  As  the  V&V 
Report  is  a  critical  document  in  the  accreditation  process,  modification  to  the 
report  might  be  necessary  to  clarify  V&V  results  or  to  correct  deficiencies.  This 
ensures  issues  can  be  resolved  or  at  least  properly  documented  prior  to  the  Test 
Readiness  Review  (TRR)  and  Accreditation  Decision.  After  evaluation  of  the 
Assessment  Report,  the  SAT  ESG  makes  an  accreditation  recommendation  to 
the  AA. 

2.3.4  Phase  4:  Accreditation 

The  AA  or  their  designee  conducts  the  TRR  to  determine  if  the  test 
planning  is  complete  and  to  assess  the  risk  associated  with  using  the  M&S. 
When  the  AA  signs  the  Test  Plan  and  approves  conducting  the  test/analysis, 
they  simultaneously  accredit  the  M&S.  The  AA  can  also  approve  a  limited 
accreditation,  request  additional  information,  or  deny  the  accreditation. 

The  accreditation  remains  in  effect  as  long  as  the  intended  use  or 
limitations/assumptions  of  the  M&S  do  not  change,  or  until  revoked  by  the  AA.  If 
it  is  desired  to  expand  or  change  the  intended  use  or  assumptions/limitations  of 
the  M&S  defined  in  the  M&S  Accreditation  Decision  Letter,  the  M&S  must  be 
submitted  for  re-accreditation. 
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The  MSD  conducts  the  test/analysis  per  the  Test  Plan  and  documents 
results  in  the  Test  Report.  It  is  essential  that  any  issues  related  to  M&S  be 
described  in  the  report,  especially  recommendations  for  improving  the  fidelity 
and/or  capabilities  of  the  M&S.  The  M&S  Chief  and  the  VV&A  Action  Team 
evaluate  this  information.  Lastly,  the  M&S  Chief  posts  unclassified  VV&A 
documentation  to  the  SIAP  Worksite. 
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3.  High  Level  Architecture  (HLA)/Federations  of  M&S 


The  HLA  is  a  general-purpose  architecture  for  federations  of  M&S.  The 
HLA  was  developed  under  the  leadership  of  the  Defense  Modeling  and 
Simulation  Office  (DMSO)  to  promote  reuse  and  interoperability  across  the  large 
numbers  of  different  types  of  simulations  developed  and  maintained  by  the  DoD. 
The  HLA  Baseline  Definition  was  completed  on  21  Aug  96.  The  Under  Secretary 
of  Defense  for  Acquisition  and  Technology  (USD  (A&T))  approved  it  as  the 
standard  technical  architecture  for  all  DoD  simulations  on  10  Sep  96. 

VV&A  activities  will  be  conducted  as  an  integral  part  of  the  federation 
systems  engineering  process.  VV&A  of  M&S  federations  identifies  risks  to  the 
credibility  of  the  federation  results  for  the  purposes  of  a  specific  analysis,  as  well 
as  the  actions  required  to  mitigate  those  risks.  The  SIAP  M&S  VV&A  Process 
also  applies  to  federation  accreditation. 

3.1  Federation  Development  and  Execution  Process  (FEDEP) 

The  development  of  federations  designed  to  support  TF  engineering 
efforts  follow  the  six-phase  FEDEP  process,  which  has  been  used  for  Joint 
Distributed  Engineering  Plant  (JDEP)  events.  The  phases  in  this  process  are 
shown  in  Figure  2.  The  VV&A  process  shown  in  Figure  1  is  still  applicable  to  the 
FEDEP  process,  with  the  same  documentation  and  reviews. 


Figure  2.  Federation  Development  and  Execution  Process 
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3.1.1  Phase  1:  Define  Federation  Objectives 

As  with  any  VV& A  process,  the  first  step  is  to  clearly  define  the  federation 
objectives.  Phase  1  of  the  FEDEP  process  is  equivalent  to  the  Designation  phase 
of  the  SIAP  VV&A  Process. 

3.1.2  Phase  2:  Develop  Federation  Conceptual  Model 

The  next  phase  is  to  define  characteristics  of  federates  and  the  federation 
needed  to  address  issues.  Of  particular  importance  is  credibility  of  the  scenario 
and  its  appropriateness  as  a  context  for  the  analysis  (sufficient  numbers  and 
positions  of  friendly  and  enemy  forces).  These  federation  requirements  drive  the 
selection  of  federates  and  the  VV&A  of  the  federation.  This  phase  requires 
active  participation  of  the  SMEs  and  the  MSDs,  since  it  is  dependent  on  a  sound 
understanding  of  the  problem  area,  the  substantive  issues  to  be  addressed  in  the 
test,  and  requirements  for  selection  of  the  federates  to  meet  the  needs  of  the 
test.  Phase  2  of  the  FEDEP  process  occurs  during  the  Planning  phase  of  the 
SIAP  VV&A  Process. 

3.1.3  Phase  3:  Design  Federation 

The  next  phase  is  to  identify  specific  federates,  develop  the  Federation 
Object  Model  (FOM),  and  delineate  federate  upgrades  to  support  federation. 

The  federation  design  reflects  the  decision  of  how  to  satisfy  the  federation 
requirements  with  specific  federates,  scenarios  and  data  exchanges.  It  may  be 
necessary  to  review  the  objectives  for  clarity,  and  return  to  the  conceptual 
analysis  with  more  detail  to  ensure  the  requirements  for  the  federation  are  well 
articulated  and  understood.  Phase  3  of  the  FEDEP  process  also  occurs  during 
the  Planning  phase  of  the  SIAP  VV&A  Process. 

3.1.4  Phase  4:  Develop  Federation 

Next,  federate  owners  implement  support  for  the  FOM  and  enhancements 
in  federates  as  needed  and  test  individual  federates.  This  is  done  by  the  owners 
of  the  federates  and  the  federation  developers.  Phase  4  of  the  FEDEP  process 
occurs  during  the  V&V  Execution  phase  of  the  SIAP  VV&A  Process. 

3.1.5  Phase  5:  Integrate  and  Test  Federation 

Incremental  testing  of  federation  capabilities  and  sets  of  federates  is 
completed  to  prepare  for  the  federation  execution  to  support  the  test.  Phase  5  of 
the  FEDEP  process  also  occurs  during  the  V&V  Execution  phase  of  the  SIAP 
VV&A  Process. 
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3.1.6  Phase  6:  Execute  Federation  and  Prepare  Results 

A  TRR  is  held  and  the  test  is  conducted  using  the  federation  following  the 
test  process  and  procedures.  Phase  6  of  the  FEDEP  process  occurs  during  the 
Accreditation  phase  of  the  SIAP  VV&A  Process. 

Table  2.  FEDEP/SIAP  VV&A  Mapping  Chart 
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6. 


GLOSSARY  OF  TERMS 


ACCREDITATION:  The  official  certification  that  a  model,  simulation,  or 
federation  of  models  and  simulations  and  its  associated  data  are  acceptable  for 
use  for  a  specific  purpose. 

ACCREDITATION  AUTHORITY:  The  government  organization  that  is 
responsible  for  the  results  from  usage  of  the  designated  M&S.  The  accreditation 
authority  shall  approve  designated  M&S  for  VV&A  and  accredit  the  M&S. 

DATA:  A  representation  of  facts,  concepts,  or  instructions  in  a  formalized 
manner  suitable  for  communication,  interpretation,  or  processing  by  humans  or 
by  automatic  means. 

FEDERATE:  An  individual  model  or  simulation  that  is  part  of  a  federation  of 
models  and  simulations. 

FEDERATION:  A  system  of  interacting  models  and/or  simulations,  with 
supporting  infrastructure,  based  on  a  common  understanding  of  the  objects 
portrayed  in  the  system. 

MODEL:  A  physical,  mathematical,  or  otherwise  logical  representation  of  a 
system,  entity,  phenomenon,  or  process. 

M&S  DEVELOPER:  An  individual,  group,  or  organization  responsible  for  actually 
developing  or  modifying  a  simulation  in  accordance  with  design  requirements 
and  specifications.  The  MSD  provides  the  required  V&V  information,  conducts 
V&V  activities,  and  supports  the  SAT  ESG  as  tasked. 

M&S  USER:  The  individual,  group,  or  organization  that  employs  or  will  employ  a 
model,  simulation,  or  simulation  federation,  its  products,  or  its  services  to  achieve 
a  set  of  objectives. 

SIMULATION:  A  method  for  implementing  a  model  over  time. 

SUBJECT  MATTER  EXPERT:  An  individual  who,  by  virtue  of  position, 
education,  training,  or  experience,  is  expected  to  have  greater-than-normal 
expertise  or  insight  relative  to  a  particular  technical  or  operational  discipline, 
system,  or  process.  They  may  be  an  individual  from  the  Program  Office,  a 
contractor,  a  field  activity  or  a  laboratory. 

VALIDATION:  The  process  of  determining  the  degree  to  which  a  model 
(simulation)  is  an  accurate  representation  of  the  real  world  from  the  perspective 
of  the  intended  uses  of  the  model. 
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VERIFICATION:  The  process  of  determining  that  a  model  (simulation) 
implementation  accurately  represents  the  developer’s  conceptual  description  and 
specifications. 
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7.  ACRONYMS 

Acronym 

Definition 

AA 

Accreditation  Authority 

BMDO 

Ballistic  Missile  Defense  Office 

CM 

Configuration  Management 

DMAP 

Data  Management  and  Analysis  Plan 

DMSO 

Defense  Modeling  and  Simulation  Office 

DoD 

Department  of  Defense 

HWIL 

Hardware-in-the-Loop 

IAP 

Integrated  Assessment  Plan 

JROC 

Joint  Requirements  Oversight  Council 

M&S 

Modeling  and  Simulation 

MSD 

Modeling  and  Simulation  Developer 

OSD 

Office  of  the  Secretary  of  Defense 

POA&M 

Plan  of  Action  and  Milestones 

RPG 

Recommended  Practices  Guide 

SAT  ESG 

SIAP  Analysis  Team  Executive  Steering  Group 

SETF 

System  Engineering  Task  Force 

SIAP 

Single  Integrated  Air  Picture 

SME 

Subject  Matter  Expert 

TRR 

Test  Readiness  Review 

V&V 

Verification  and  Validation 

VV&A 

Verification,  Validation,  and  Accreditation 
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APPENDIX.  SAMPLE  PRODUCT  FORMATS 
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DEPARTMENT  OF  DEFENSE 

SINGLE  INTEGRATED  AIR  PICTURE 

system  engineering  task  force 

1931  JEFFERSON  DAVIS  HIGHWAY 
CM3  SUITE  1142 

ARLINGTON  VA  22202-3523  IN  heply  refer  to 

9800 

Ser  SIAP/072 
19  Sep  02 

From:  Single  Integrated  Air  Picture  System  Engineer 

To:  Center  for  Naval  Analyses  (Attn:  Mr.  Paul  Symborski) 

Subj  :  DESIGNATION  OF  OPERATIONAL  DATA  DRIVEN  SIMULATION  FOR 

CORRELATION  ALGORITHM  PERFORMANCE  EVALUATION  (ODDSCAPE) 
IN  SUPPORT  OF  SINGLE  INTEGRATED  AIR  PICTURE  (SIAP)  FOR 
VERIFICATION,  VALIDATION  AND  ACCREDITATION  (W&A) 

Ref:  (a)  DoDINST  5000.61,  DoD  Modeling  and  Simulation  (MAS) 

Verification,  Validation,  and  Accreditation  (W&A) 

Enel:  (1)  Draft  SIAP  Technical  Report  2002-013,  SIAP 

Accreditation  Process  for  M&S  (in  signature  process) 
(2)  SIAP  Technical  Report  2002-005,  SIAP  Integrated 
Assessment  Plan  (IAP) 

1.  Summary.  This  letter  initiates  the  SIAP  system  Engineering 
Task  Force's  accreditation  of  the  Center  of  Naval  Analysis 
(CNA's)  ODDSCAPE  SIAP  Block  1  Phase  1  analysis.  We  expect  to 
accredit  this  model  for  perturbation  analyses  addressing 
correlation/decorrelations  algorithm  (ICP)  parameter 
optimization,  Track  quality  Sensitivity  (TQ)  corrections,  and 
inactive  Precise  Participant  Location  and  Identification  (PLI) 
logic,  (specified  in  Appendix  C  of  enclosure  (2)) - 


2,  Background.  Reference  (a)  establishes  procedures  and 
guidance  for  the  use  of  Modeling  and  Simulation  (M&S)  used  to 
support  major  DoD  organizations  and  processes,  including  the 
Joint  Requirements  Oversight  Council.  Enclosure  (1)  provides 
specifically  tailored  W&A  guidance  for  the  SIAP  Blockl  Phase  1. 
Enclosure  (2)  is  the  primary  planning  document  for  SIAP  Block 
System  Engineering  Analyses . 


3.  Action.  The  SIAP  SE  needs  your  continued  CNA  support  for  the 
SIAP  Analysis  Team  Executing  Steering  Group  (SAT  ESG)  W&A 
effort.  The  SAT  ESG  shall  coordinate  this  effort  and  should  be 
contacted  if  there  are  any  questions  regarding  M&S  accreditation. 
The  POC  for  W&A  within  the  SIAP  SE  Task  Force  is  CDR  Paul 
Votruba.  He  can  be  reached  at  (703)  602-6441.  x217,  or  by  email 
at  votrubapm@navsea.navy.mil. 


H.  V.  DUTCHY5HYN 
By  direction 
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DEPARTMENT  OF  DEFENSE 

SINGLE  INTEGRATED  AIR  PICTURE 
SYSTEM  ENGINEERING  TASK  FORCE 
1931  JEFFERSON  DAVIS  HIGHWAY 
CM3  SUITE  1142 

ARLINGTON  VA  22202-3523  in  nsf’iv  refer  to 

9800 

Ser  SIAP/008 
14  Feb  03 

From:  Single  Integrated  Air  Picture  System  Engineer 

To:  Center  for  Naval  Analyses  (Attn:  Mr.  Paul  Symborski) 

Subj :  ACCREDITATION  OF  OPERATIONAL  DATA  DRIVEN  SIMULATION  FOR 

CORRELATION  ALGORITHM  PERFORMANCE  EVALUATION  ( ODDSCAPE ) FOR 
USE  IN  ANALYSES  OF  ASCIET  00  AND  JCIET  02  DATA 


Ref  : 


(a)  Memo  For  Record,  SIAP  Analysis  Team  (SAT)  Executive 
Steering  Group  Chairman,  6  Feb  03 

(b)  Verification  and  Validation  Report  for  ODDSCAPE, 

CNA,  7  Nov  02 

(c)  ODDSCAPE  Designation  Letter,  9800  Ser  SIAP/072, 

19  Sep  02 

(d)  DoDINST  5000.61,  DoD  Modeling  and  Simulation 

(M&S)  Verification,  Validation  and  Accreditation 
(W&A) 


1.  Based  on  the  information  contained  in  references  (a)  through 
(d) ,  the  Center  for  Naval  Analyses'  ODDSCAPE  is  accredited  for 
use  to  support  SIAP  Block  1  analyses.  Specific  use  is  to  be 
limited  to  perturbation  analyses  addressing  Correlation/ 
Decorrelation  algorithm  (Interface  Change  Proposal)  parameter 
optimization,  Track  Quality  (TQ)  sensitivity  corrections, 
inactive  Precise  Participant  Location  and  Identification  (PPLI) 
logic  and  Data  Registration.  This  accreditation  covers  the 
analysis  of  data  from  ASCIET  00  and  JCIET  02. 


2.  This  accreditation  applies  only  to  ODDSCAPE  Version  2.0. 
Major  changes  to  this  model's  configuration  and/or  a  change  of 
specific  use  not  addressed  here  should  be  separately  accredited 
through  this  office. 


3.  My  point  of  contact  for  W&A  is  Maj  David  Borowsky, 
(703)602-6441  x265  or  by  email  at  borowskydj@navsea.navy.mil. 


H .  V .  DUTCHYSHYN 
By  direction 


Copy  To : 

NSWC  Corona  (Mr.  Dan  Bergstrom) 
PEO  AMD  (Mr.  Ray  Washburn) 
MARCORSYSCOM  (Maj  Madsen) 

ESC/DI  (Mr.  Basel  Brown) 
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Sample  Configuration  Management  Plan 
Scope 

Reference  Documents 

Configuration  Management  Environment 

Configuration  Control  Process 

Sample  V&V Plan 

Executive  Summary 
Model  Description 

M&S  Requirements  and  Acceptance  Criteria 

V&V  Approach 

M&S  V&V  History 

Schedule 

References 

Sample  V&  V  Report 

Executive  Summary 
Differences  from  V&V  Plan 
V&V  Results 
V&V  Summary 

Sample  V&V  Assessment  Report 

1 .  Summary 

2.  Application  M&S  Requirements  and  Acceptance  Criteria 

3.  Model  Capability 

4.  V&V  Report  Summary 

5.  Comparison  Analysis  of  Requirements  Versus  Capabilities 

6.  Comparison  Analysis  Summary 

7.  Other  Accreditation  Information 

8.  Recommendations 
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